Opinion Marks: A Human-Based Computation Approach to Instill Structure into Unstructured Text on the Web

Despite recent improvements in various computational approaches such as machine learning, natural language processing, and computational linguistics, making a computer understand unstructured, human-generated text still remains a difficult problem to solve. To alleviate the challenges, we propose an approach called “Opinion Marks,” which enables writers to mark positive and negative aspects of a topic on their own text. In addition, Opinion Marks incorporates an automatic marking suggestion algorithm to offload a user’s marking effort. The phrases marked with Opinion Marks can be further used to clarify sentiments of other text in the similar context. We implemented Opinion Marks on a question answering website http://caniask.net. To test the efficacy of Opinion Marks, we conducted a crowdsourcing-based experiment with 144 participants in a between-subject design under the three different conditions: 1) human marking only; 2) machine marking only (automatic marking suggestion); and 3) human-machine collaboration (Opinion Marks). This study revealed that Opinion Marks significantly improves the quality of marked phrases and usability of the system.